Method for combined transformation of the scale and aspect ratio of a picture

ABSTRACT

A source image is transformed into a destination image having a target aspect ratio. A reference region in the source image is defined. An extended region of interest of the source image having the target aspect ratio and containing the reference region is defined. A set of candidate image regions of increasing resolutions from the extended region of interest is determined, each having the target aspect ratio and containing the reference region. Candidate image regions are scaled to form a candidate target images. A quality metric is used to select a target image providing the best quality metric value.

BACKGROUND

Technical Field

This disclosure relates to the rendering of images on displays, or ondisplay areas having a resolution and/or an aspect ratio differing fromthose of the original image.

Description of the Related Art

Given the numerous image formats (still or video), with a variety ofresolutions and aspect ratios, it often happens that a display is notadapted to the original image, for example when viewing a 4/3 video on a16/9 display, or vice versa. The formats for still pictures are evenmore varied, whereby more difficulties are encountered.

In some applications, it is desired to display a mosaic of images, suchas thumbnails of a photo album, or sub-pictures often calledPicture-In-Picture (PIP) from multiple sources. Thumbnails orsub-pictures may have an aspect ratio close to 1, while the sources mayhave an aspect ratio of 16/9 or 4/3, or other ratios, such as forphotos.

FIG. 1 illustrates two conventional rendering techniques that may beused to display a 16/9 image (left) in a square thumbnail (right), forexample.

In the upper thumbnail, the original image's aspect ratio is preserved.The scaling of the original image is calculated to match the largestdimension of the image to a side of the square. The modified image iscentered vertically in the square, and black bands are displayed in theareas of the square corresponding to no image. Part of the thumbnail isnot used to display the image.

In the lower thumbnail, the image is transformed to fill the square. Forthis purpose, the vertical and horizontal dimensions of the originalimage undergo different scaling factors. The entire surface of thethumbnail is used to display the image, but the image has a verticalstretch distortion.

Moreover, visual quality deterioration may often be perceived in thethus processed images, even when using sophisticated scaling algorithms.

BRIEF SUMMARY

In an embodiment, a method of transforming a source image in adestination image having a target aspect ratio, comprises defining areference region in the source image; finding a minimal zone of thesource image having the target aspect ratio and containing the referenceregion; forming a set of candidate image regions of increasingresolutions from the minimal region, each having the target aspect ratioand containing the reference region; scaling each candidate region toform a candidate target image; calculating a quality metric on eachcandidate target image; and selecting for the final target image, thecandidate target image providing the best quality metric value.

The reference region may be defined by a region-of-interest searchalgorithm.

The method may comprise storing a list of unfavorable scaling factors;and when forming the set of candidate image regions, omitting candidateregions producing any of the unfavorable scaling factors in forming thetarget image.

The method may comprise calculating a default quality metric on eachcandidate target image; if the default quality metric does not produce adiscriminating value on the set of candidate target images, calculatinga second quality metric for each candidate target image; and selectingthe candidate target image providing the best value for the secondquality metric.

The quality metric may be blur metric, such as a blur metrix based onlyon the target image.

The quality metric may be a structural similarity factor based on thecandidate image region and the corresponding candidate target image.

The default quality metric may be a blur metric based only on the targetimage, and the second quality metric may be a structural similarityfactor based on the candidate image region and the correspondingcandidate target image.

In an embodiment, a method comprises: determining a reference region ofa source image; determining an extended region of interest of the sourceimage, the extended region of interest having a target aspect ratio andcontaining the reference region; determining a set of candidate imageregions including regions of increasing size from the extended region ofinterest, each candidate image region having the target aspect ratio andcontaining the reference region; scaling candidate image regions to forma set of candidate target images; determining quality metric values ofimages of the set of candidate target images; and selecting one of thecandidate target images of the set of candidate target images as atarget image based on the determined quality metric values. In anembodiment, the method comprises using a region-of-interest searchalgorithm to determine the reference region. In an embodiment, themethod comprises storing a list of unfavorable scaling factors; andexcluding candidate regions producing any of the unfavorable scalingfactors from the set of candidate target images. In an embodiment, themethod comprises: determining a default quality metric value for eachcandidate target image of the set of candidate target images; and whenthe determined default quality metric values do not satisfy an imagequality criteria, calculating a second quality metric value for eachcandidate target image of the set of candidate target images; andselecting one of the candidate target images of the set of candidatetarget images based on the determined second quality metric values. Inan embodiment, the quality metric is a blur metric based only on acandidate target image. In an embodiment, the quality metric is astructural similarity factor (SSIM) based on a candidate image regionand corresponding candidate target image. In an embodiment, the defaultquality metric is a blur metric based only on a candidate target image,and the second quality metric is a structural similarity factor based ona candidate image region and a corresponding candidate target image. Inan embodiment, the set of candidate image regions includes the extendedregion of interest. In an embodiment, the reference region includes aplurality of regions of interest.

In an embodiment, a device comprises: one or more memories; and one ormore processing devices which, in operation, determine an extendedregion of interest of a source image, the extended region of interesthaving a target aspect ratio and containing a reference region of thesource image; determine a set of candidate image regions of the sourceimages, the set of candidate image regions including regions ofincreasing size from the extended region of interest, each candidateimage region having the target aspect ratio and containing the referenceregion; scale candidate image regions to form a set of candidate targetimages; determine quality metric values of images of the set ofcandidate target images; and select one of the candidate target imagesof the set of candidate target images as a target image based on thedetermined quality metric values. In an embodiment, the one or moreprocessing device, in operation, determine the reference region using aregion of interest search algorithm. In an embodiment, in operation, theone or more memories store a list of unfavorable scaling factors; andcandidate regions producing any of the unfavorable scaling factors areexcluded from the set of candidate target images. In an embodiment, inoperation, the one or more processing devices: determine a defaultquality metric value for each candidate target image of the set ofcandidate target images; and when the determined default quality metricvalues do not satisfy an image quality criteria, determine a secondquality metric value for each candidate target image of the set ofcandidate target images; and select one of the candidate target imagesof the set of candidate target images based on the determined secondquality metric values. In an embodiment, the quality metric is a blurmetric based only on a candidate target image. In an embodiment, thequality metric is a structural similarity factor (SSIM) based on acandidate image region and corresponding candidate target image. In anembodiment, the default quality metric is a blur metric based only on acandidate target image, and the second quality metric is a structuralsimilarity factor based on a candidate image region and a correspondingcandidate target image. In an embodiment, the set of candidate imageregions includes the extended region of interest. In an embodiment, thereference region includes a plurality of regions of interest.

In an embodiment, a system comprises: means for determining an extendedregion of interest of a source image, the extended region of interesthaving a target aspect ratio and containing a reference region of thesource image; means for determining a set of candidate image regions ofthe source images, the set of candidate image regions including regionsof increasing size from the extended region of interest, each candidateimage region having the target aspect ratio and containing the referenceregion; means for scaling candidate image regions to form a set ofcandidate target images; means for determining quality metric values ofimages of the set of candidate target images; and means for selectingone of the candidate target images of the set of candidate target imagesas a target image based on the determined quality metric values. In anembodiment, the system comprises: means for determining the referenceregion. In an embodiment, the system comprises: means for excludingcandidate target regions having one of a set of scaling factors from theset of candidate target images. In an embodiment, the system comprises:means for selecting a quality metric.

In an embodiment, a non-transitory computer-readable medium's contentscause a computing device to perform a method, the method comprising:determining an extended region of interest of a source image, theextended region of interest having a target aspect ratio and containinga reference region; determining a set of candidate image regionsincluding regions of increasing size from the extended region ofinterest, each candidate image region having the target aspect ratio andcontaining the reference region; scaling candidate image regions to forma set of candidate target images; determining quality metric values ofimages of the set of candidate target images; and selecting one of thecandidate target images of the set of candidate target images as atarget image based on the determined quality metric values. In anembodiment, the method comprises using a region-of-interest searchalgorithm to determine the reference region. In an embodiment, themethod comprises: determining a default quality metric value for eachcandidate target image of the set of candidate target images; and whenthe determined default quality metric values do not satisfy an imagequality criteria, calculating a second quality metric value for eachcandidate target image of the set of candidate target images; andselecting one of the candidate target images of the set of candidatetarget images based on the determined second quality metric values.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

Other potential advantages and features of various embodiments willbecome more clearly apparent from the following description ofparticular embodiments provided for exemplary purposes only andrepresented in the appended drawings, in which:

FIG. 1, previously described, shows a source image transformed using twodifferent conventional techniques to be displayed with an aspect ratiodifferent from that of the source image; and

FIGS. 2A through 2C illustrate different steps of an imagetransformation method offering improved visual quality.

DETAILED DESCRIPTION

An embodiment of an image processing method is described below inconnection with the example of FIG. 1, where a 16/9 source image istransformed in a target image that may be displayed in a square area. Ina general situation, the resolutions of the source and target images aredifferent and the aspect ratios of these images are different andarbitrary.

FIG. 2A shows the same source image as FIG. 1 having an aspect ratio of16/9. It is desired to transform this image, in this example, to renderit in a square area, having an aspect ratio of 1/1. The square area maybe a photo album thumbnail or a sub-picture of a mosaic of TV channels.In this case, the resolution of the square area is generally lower thanthe resolution of the source image. In another example, the display areamay have a higher resolution than the source image, for example when lowresolution videos are displayed on an HD screen.

As noted above, it is desired to transform the source image to render itin an “intelligible” manner in the display area. In an embodiment, onlya portion containing key information of the source image is displayed,having dimensions chosen to provide the best visual quality.

For this purpose, a reference region ROI is first defined in the sourceimage, whose aspect ratio may be arbitrary. The reference region may bedefined manually by the user, but it may be provided automatically by aconventional Region-Of-Interest (ROI) search algorithm.

Face recognition is a typical application of such an algorithm. In theexample of FIG. 2A, a reference region ROI is defined around a face. Asshown, this region ROI is rectangular with an aspect ratio close to 3/4,differing from the aspect ratio of the source image, but also differingfrom the aspect ratio of 1/1 of the display area.

Generally, region-of-interest search algorithms produce rectangularregions of arbitrary aspect ratio. They may also produce several regionsof interest, for example in the presence of multiple faces.

In FIG. 2B, an extended region of interest ROIe is defined, which hasthe aspect ratio of the target display (1/1 here), and may have asmallest size able to fully contain the reference area (hatched in thefigure), or reference areas in the case where multiple regions ofinterest are identified. In this example, the extended region ROIe is asquare whose upper and lower sides coincide with the upper and loweredges of the reference region ROI. The remaining, left and right sidesmay be, for example, at equal distances from the left and right edges ofthe reference region.

In a general situation, the reference region ROI is comprised within thesource image such that there is a margin between each of the edges ofthe region ROI and the corresponding edge of the source image, as shown.If the margin is insufficient for centering the extended region ROIearound the reference region, such as when the reference region sitsagainst an edge of the source image, the extended region may beuncentered with respect to the reference region in order to maintain thetarget aspect ratio.

In other words, if the margins around the reference region ROI allow it,the extended region ROIe may be centered about the reference region. Ifthe margins are insufficient, the extended region may be shifted so asto maintain its aspect ratio while containing the reference region. In aborderline case example, in FIG. 2B, the right edge of the region ROImay coincide with the right edge of the source image. Then the extendedregion ROIe is shifted to the left so that its right edge coincides withthe right edge shared between region ROI and the source image.

A region-of-interest search algorithm may produce several regions, forexample in the presence of multiple faces. The extended region ROIe maythen be defined, for example, to contain all the identified regions ofinterest, only the largest one, or the one closest to the center of thesource image, a selected one of the regions of interest, etc.

Once the extended region ROIe is thus defined, it could be sufficient toscale it, homogeneously vertically and horizontally, to produce a targetimage compatible with the display area, e.g., that fills the displayarea without distortion.

However, it appears that some scaling factors, depending on the imagecontent and the scaling algorithm, produce visual effects that affectthe intelligibility or the perceived quality of the final image.Intelligibility is altered, for example, when the image contains finerepetitive structures, especially if they are in motion. The perceivedquality is generally altered by a lower sharpness or sharpnessvariations of a texture. For example, when using bilinear and bicubicscaling algorithms, scale factors not equal to 1 comprised between 0.9and 1.1 produce poor results in most cases. But other undesired valuesmay exist that depend on image content.

FIG. 2C illustrates a series of additional steps for producing a targetimage of good visual quality in an automated manner. Based on theextended region ROIe, hatched in FIG. 2C, a set of candidate imageregions C0, C1, C2, . . . are formed that all contain the extendedregion ROIe and have the same aspect ratio (here 1/1). The firstcandidate region C0 is the extended region ROIe. The candidate regionsgrow from one to the next and, may remain centered on the initial regionC0.

Several scaling factors for forming the target image are evaluated, andone that produces the best visual quality is selected, for example,according to an evaluation criteria.

A current candidate region may be defined by expanding the smallerdimension of the previous candidate region by an increment in number ofpixels on either side. The larger dimension is adjusted, to the nearestpixel, to preserve the aspect ratio and centering. If the aspect ratiois 1/1, the previous candidate region may be expanded by one incrementfrom each edge.

At one point, one of the edges of the current candidate region may reachthe corresponding edge of the source image (the lower edge in FIG. 2C).In this case, the subsequent candidate regions may be enlarged in theremaining margin at the opposite side (upwards in FIG. 2C) and may beoff-centered relative to the initial region C0.

The set of candidate regions may be completed when, as shown for regionC2, the current candidate region reaches two opposite edges of thesource image, and can no longer grow preserving the aspect ratio, when athreshold number of candidate regions is obtained, etc. In practice, aset of less than ten candidate regions is generally sufficient and thetwo edges of the image are reached only when the margins between thereference region and the edges of the source image are too small.

From each candidate region Ci a candidate target image Ti is formed byapplying to the candidate region, at 10, the scaling factor Zi requiredto reach the resolution defined for the target image. On each candidatetarget image thus obtained a quality metric Q-METRIC is calculated. Bycomparing, in 12, the values Q produced for the quality metric, acandidate region Cs and corresponding scaling factor Zs may be selectedthat produce the best quality metric value Q.

As indicated above, there are scaling factors that should almost alwaysbe avoided, regardless of the content of the source image and thescaling algorithm, for example factors not equal to 1 comprised between0.9 and 1.1. There may be other unwanted factors that are independent ofthe content of the image but that depend on the scaling algorithm.

By storing a list of unwanted scaling factors, the method may bedesigned to exclude from the set to evaluate the candidate regionsassociated with scaling factors contained in the list. This reduces thenumber of candidates to assess to candidates that are likely to beselected.

Scaling factors above an upper threshold may be among the unwantedfactors, because they lead to displaying in high-resolution an imagearea with too little information. If the reference region ROI (FIG. 2A)turns out to be too small, requiring a scaling factor above thethreshold, the method may be designed to expand the extended referenceregion ROIe (FIG. 2B), and therefore the first candidate C0, to the sizeproducing the threshold scaling factor. The region ROIe is then largerelative to the region ROI; it may be centered about the region ROI.

Using this technique of selecting the extended reference region ROIe,the reference region ROI may be limited to a point of the image that theuser may define manually using a human interface, such as a touchscreen.

The quality metric may be chosen simple so that the calculations may beperformed in real time without delaying the display of the final targetimage. In this case a blur metric based solely on the target image maybe used, as described in [Frédérique Crete et al, “The Blur Effect.Perception and Estimation with a New No-Reference Perceptual BlurMetric” SPIE Electronic Imaging Symposium Conf., Human Vision andElectronic Imaging, San Jose 2007].

If computational resources are sufficient, a more sophisticated qualitymetric may be used, based on the source candidate region Ci and on thetarget image Ti, such as a structural similarity factor (SSIM) modifiedas described in US patent application 2013-0243333. Such a modified SSIMfactor enables the comparison of two images of different sizes. FIG. 2also shows an image processing system 200, which as illustrated includescircuitry, such as one or more processors P, one or more memories M anddiscrete circuitry DC (e.g., logic gates, state machines, multiplexers,look-up tables, etc.), which, in operation, implements one or moreembodiments of the methods described herein. For example, a processor Pmay execute a set of instructions stored in the memory M to perform anembodiment of one of the described methods.

In some circumstances a metric may be more discriminating than another,e.g., a first metric (e.g., the blur metric) may not distinctly reveal abetter value in a set of candidates, while the second metric (e.g., theSSIM factor) may reveal a singular value on the same set of candidates.To account for this situation, the method may be designed to calculate asecond metric on the candidates in cases where a default metric producesvalues below a threshold. The default metric may be the simplest tocalculate, which leads to calculating the more sophisticated metric onlyin exceptional circumstances.

Some embodiments may take the form of computer program products. Forexample, according to one embodiment there is provided a computerreadable medium comprising a computer program adapted to perform one ormore of the methods described above. The medium may be a physicalstorage medium such as for example a Read Only Memory (ROM) chip, or adisk such as a Digital Versatile Disk (DVD-ROM), Compact Disk (CD-ROM),a hard disk, a memory, a network, or a portable media article to be readby an appropriate drive or via an appropriate connection, including asencoded in one or more barcodes or other related codes stored on one ormore such computer-readable mediums and being readable by an appropriatereader device.

Furthermore, in some embodiments, some or all of the systems and/ormodules may be implemented or provided in other manners, such as atleast partially in firmware and/or hardware, including, but not limitedto, one or more application-specific integrated circuits (ASICs),discrete circuitry, standard integrated circuits, controllers (e.g., byexecuting appropriate instructions, and including microcontrollersand/or embedded controllers), field-programmable gate arrays (FPGAs),complex programmable logic devices (CPLDs), etc., as well as devicesthat employ RFID technology. In some embodiments, some of the modules orcontrollers separately described herein may be combined, split intofurther modules and/or split and recombined in various manners.

The various embodiments described above can be combined to providefurther embodiments. Aspects of the embodiments can be modified, ifnecessary to employ concepts of the various patents, applications andpublications to provide yet further embodiments.

These and other changes can be made to the embodiments in light of theabove-detailed description. In general, in the following claims, theterms used should not be construed to limit the claims to the specificembodiments disclosed in the specification and the claims, but should beconstrued to include all possible embodiments along with the full scopeof equivalents to which such claims are entitled. Accordingly, theclaims are not limited by the disclosure.

The invention claimed is:
 1. A method, comprising: determining areference region of a source image; determining an extended region ofinterest of the source image, the extended region of interest having atarget aspect ratio and containing the reference region; determining aset of candidate image regions including regions of increasing size fromthe extended region of interest, each candidate image region having thetarget aspect ratio and containing the reference region; scalingcandidate image regions to form a set of candidate target images;determining quality metric values of images of the set of candidatetarget images; and selecting one of the candidate target images of theset of candidate target images as a target image based on the determinedquality metric values, wherein the method includes: determining adefault quality metric value for each candidate target image of the setof candidate target images; and when the determined default qualitymetric values do not satisfy an image quality criteria, calculating asecond quality metric value for each candidate target image of the setof candidate target images; and selecting one of the candidate targetimages of the set of candidate target images based on the determinedsecond quality metric values.
 2. The method of claim 1, comprising usinga region-of-interest search algorithm to determine the reference region.3. The method of claim 1, comprising: storing a list of unfavorablescaling factors; and excluding candidate regions producing any of theunfavorable scaling factors from the set of candidate target images. 4.The method of claim 1 wherein the quality metric is a blur metric basedonly on a candidate target image.
 5. The method of claim 1 wherein thequality metric is a structural similarity factor (SSIM) based on acandidate image region and corresponding candidate target image.
 6. Themethod of claim 1 wherein the default quality metric is a blur metricbased only on a candidate target image, and the second quality metric isa structural similarity factor based on a candidate image region and acorresponding candidate target image.
 7. The method of claim 1 whereinthe set of candidate image regions includes the extended region ofinterest.
 8. The method of claim 1 wherein the reference region includesa plurality of regions of interest.
 9. The method of claim 1 wherein thedetermining quality metric values of images of the set of candidatetarget images is based on an image quality criteria.
 10. A device,comprising: one or more memories; and one or more processing deviceswhich, in operation, determine an extended region of interest of asource image, the extended region of interest having a target aspectratio and containing a reference region of the source image; determine aset of candidate image regions of the source images, the set ofcandidate image regions including regions of increasing size from theextended region of interest, each candidate image region having thetarget aspect ratio and containing the reference region; scale candidateimage regions to form a set of candidate target images; determinequality metric values of images of the set of candidate target images;and select one of the candidate target images of the set of candidatetarget images as a target image based on the determined quality metricvalues, wherein, in operation, the one or more processing devices:determine a default quality metric value for each candidate target imageof the set of candidate target images; and when the determined defaultquality metric values do not satisfy an image quality criteria,determine a second quality metric value for each candidate target imageof the set candidate target images; and select one of the candidatetarget images of the set of candidate target images based on thedetermined second quality metric values.
 11. The device of claim 10wherein the one or more processing device, in operation, determine thereference region using a region of interest search algorithm.
 12. Thedevice of claim 10 wherein, in operation, the one or more memories storea list of unfavorable scaling factors; and candidate regions producingany of the unfavorable scaling factors are excluded from the set ofcandidate target images.
 13. The device of claim 10 wherein the qualitymetric is a blur metric based only on a candidate target image.
 14. Thedevice of claim 10 wherein the quality metric is a structural similarityfactor (SSIM) based on a candidate image region and correspondingcandidate target image.
 15. The device of claim 10 wherein the defaultquality metric is a blur metric based only on a candidate target image,and the second quality metric is a structural similarity factor based ona candidate image region and a corresponding candidate target image. 16.The device of claim 10 wherein the set of candidate image regionsincludes the extended region of interest.
 17. The device of claim 10wherein the reference region includes a plurality of regions ofinterest.
 18. The device of claim 10 wherein the one or more processingdevices, in operation, determine quality metric values of images of theset of candidate target images based on an image quality criteria.
 19. Asystem, comprising: means for determining an extended region of interestof a source image, the extended region of interest having a targetaspect ratio and containing a reference region of the source image;means for determining a set of candidate image regions of the sourceimages, the set of candidate image regions including regions ofincreasing size from the extended region of interest, each candidateimage region having the target aspect ratio and containing the referenceregion; means for scaling candidate image regions to form a set ofcandidate target images; means for determining quality metric values ofimages of the set of candidate target images; and means for selectingone of the candidate target images of the set of candidate target imagesas a target image based on the determined quality metric values, whereinthe means for determining quality metric values of images of the set ofcandidate target images, in operation, determines a default qualitymetric value for each candidate target image of the set of candidatetarget images; and when the determined default quality metric values donot satisfy an image quality criteria, calculates a second qualitymetric value for each candidate target image of the set of candidatetarget images; and selects one of the candidate target images of the setof candidate target images based on the determined second quality metricvalues.
 20. The system of claim 19, comprising: means for determiningthe reference region.
 21. The system of claim 19, comprising: means forexcluding candidate target regions having one of a set of scalingfactors from the set of candidate target images.
 22. The system of claim19, comprising: means for selecting a quality metric.
 23. Anon-transitory computer-readable medium whose contents cause a computingdevice to perform a method, the method comprising: determining anextended region of interest of a source image, the extended region ofinterest having a target aspect ratio and containing a reference region;determining a set of candidate image regions including regions ofincreasing size from the extended region of interest, each candidateimage region having the target aspect ratio and containing the referenceregion; scaling candidate image regions to form a set of candidatetarget images; determining quality metric values of images of the set ofcandidate target images; and selecting one of the candidate targetimages of the set of candidate target images as a target image based onthe determined quality metric values, wherein the method comprises:determining a default quality metric value for each candidate targetimage of the set of candidate target images; and when the determineddefault quality metric values do not satisfy an image quality criteria,calculating a second quality metric value for each candidate targetimage of the set of candidate target images; and selecting one of thecandidate target images of the set of candidate target images based onthe determined second quality metric values.
 24. The medium of claim 23wherein the method comprises using a region-of-interest search algorithmto determine the reference region.
 25. The medium of claim 23 whereinthe determining quality metric values of images of the set of candidatetarget images is based on an image quality criteria.